👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

Why not just use mmmu_val instead of the literature task? This gives us around 0.53 for the dense model?

mmmu_val is 900 evals total instead of 30. that would add probably ~40 minutes to each lm-eval run, and we run two for each config, so total test time would increase over 3 hours with that change

Why not use mmmu_val

Why not just use mmmu_val instead of the literature task? This gives us around 0.53 for the dense model?

mmmu_val is 900 evals total instead of 30. that would add probably ~40 minutes to each lm-eval run, and we run two for each config, so total test time would increase over 3 hours with that change

The 30 datapoints has proven to be very noisy historically. A happy medium might be better but we should also just validate the runtime for batch size of 100

LGTM. 3 questions:

1. Do we need to keep all the datasets in the testing_utils? Are thre some there that we can remove?

2. Is 100 enough?

3. Did we mention to MLR the variation we see without the chat_template?

@dsikka thanks, see responses below:

1. we could prune the code in testing_utils. should i get rid of gsm8k, open-platypus and slim-orca?
2. I can up this as well. I set it to 100 because tests were taking forever, but i think the cpu of the cluster was just under heavy load when i was trying. 500?
3. I can do so Monday

LGTM. 3 questions:

1. Do we need to keep all the datasets in the testing_utils? Are thre some there that we can remove?

2. Is 100 enough?

3. Did we mention to MLR the variation we see without the chat_template?

@dsikka thanks, see responses below:

1. we could prune the code in testing_utils. should i get rid of gsm8k, open-platypus and slim-orca?
2. I can up this as well. I set it to 100 because tests were taking forever, but i think the cpu of the cluster was just under heavy load when i was trying. 500?
3. I can do so Monday

Yeah I think we shoud up to 500 and remove any testing dataset that we're not using